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ABSTRACT 

This paper summarizes the results of an analysis of the impacts of adopting information and communication technology 
(ICT) solutions in a learning context. The analysis is based on a literature survey of articles reporting research cases 
studying the impact of adopting ICT based solutions in various learning contexts. The subject has been reviewed using 
the term blended learning that can incorporate both e-leaming and m-learning approaches, as the boundaries between the 
various ICT solutions have become increasingly blurred. The focus of the study is on adult learning contexts. The 
contribution of this paper is to summarize how researchers evaluate and show the impact of adopting ICT in learning, and 
what the critical factors contributing to the success of ICT based solutions in learning are. 
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1. INTRODUCTION 

During the recent years, there has been a lot of public debate on adopting information and communication 
technology (ICT) in learning. Big ICT companies, such as IBM, Apple and Microsoft, have launched 
initiatives aiming at bringing ICT to schools, universities and other learning contexts. The education system 
is investing money and effort on providing teachers and learners with digital content, communication devices 
and digitally enhanced learning environments. 

The adoption of ICT based solutions for learning is not simple. Technical devices require investments, 
and they can become obsolete quickly. Developing digital content and changing teaching, learning and 
management practices takes time and resources. In this paper, we aim to explore how the current body of 
research knowledge supports decision makers in getting information on the impact of ICT based solutions on 
learning. We will contribute to this through exploring the following questions through a literature survey: 

■ How do researchers evaluate the impact of adopting ICT in learning? 

■ According to studies, what kinds of factors contribute to the impact of ICT based solutions in learning? 

In order to conduct the search for material related to the study, it was necessary to define the terminology. 

The following definitions will be used in this article: Blended learning is used to describe learning systems 
that combine traditional face-to-face instruction with computer-mediated instruction (Graham, 2006). E- 
learning is instruction that is delivered on a digital device such as a computer or a mobile device (e.g. mobile 
phone or tablet) whose purpose is to support learning (Clark & Mayer, 2008). M-learning has to provide the 
means to study and learn anywhere and at any given time, using a mobile device, and with no permanent 
physical connection to a network (Gregoriev et ah, 2004). 

Using the definitions above it can be seen that e-leaming is often used as a general term. We can conclude 
that m-leaming can be part of e-leaming, and blended learning can incorporate m-leaming and e-leaming 
approaches. For the purpose of this study, and in this article, the term blended learning is used, because it 
encompasses the necessary elements. 
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2. METHODS AND MATERIAL 

We collected research material through the means of a literature review (Webster & Watson, 2002). The 
focus was on articles written after the year 2000. The articles were collected mostly through searches in 
Google Scholar and subsequent electronic journals and databases, looking for articles combining one of the 
following terms: elearning, e-learning, blended learning, technology enhanced learning, mobile learning, 
mlearning or m-learning with impact, case study, or field trial. The study progressed through the reference 
lists of relevant articles. 

The analysis revealed that there was a dearth of scholarly articles offering solid facts and figures about the 
Return of Investment (ROI) of e-leaming solutions in workplace settings. It was easier to uncover articles 
and case studies about e-leaming or blended learning experiments in higher education. Even in these cases, 
the focus was usually on the learning outcome or participant (student or instmctor) satisfaction. 

Material assembled for this study includes also articles highlighting benefits of blended learning solutions 
other than cost savings. As the financial impact is often difficult to measure - and because it is not always 
straightforward, as savings can result from a multitude of factors - it is important to look also at the other 
perceived advantages blended learning has to offer. 


3. BENEEITS OE BLENDED LEARNING 

According to Osguthorpe & Graham (2003), the main goals for instractors designing blended learning 
environments are pedagogical richness, access to knowledge, social interaction, personal agency, cost 
effectiveness and ease of revision. In other words, these are the main characteristics that a successful blended 
learning system should have. 

Bersin (2004) states that in the early stages of e-leaming many programs failed. One of the reasons was 
that it was too easy to disengage, as there was no class to go to and no time set aside for leaming-it is 
difficult to include an extra part into an already very busy day. The solution is a blended learning program 
and finding the right blend. 

Snipes (2010) also reminds that there is not just the one right solution, and much of the success of 
blended learning is attributed to putting the building blocks of a company’s learning content together so that 
the combination results in a highly engaging learning environment. Real learning does not come from a one- 
time training event; it requires a longer learning process. In order to really impact the learners, the experience 
needs to be engaging and personal. The goal is a long-lasting business impact, not just a transient effect. 

To get the most out of an e-leaming or blended learning solution, it is best to use an IT architecture that 
supports targeting and personalization. Thus, training efforts can be directed at the employees and 
competencies that have the biggest need for learning and knowledge. (Ley & Ulbrich, 2002) 

Table 1 summarizes the benefits of blended or mobile learning reported by researchers. The studies that 
are based on the expectations of the studied group are marked in italics; the rest of the articles are based on 
case studies and field trials. 


Table 1. Main benefits of blended learning 


Benefit 

Mentioned by 

Ability to return to material and 
revise 

§ahin (2010b); Schooley (2009) 

Adaptability, tailoring 

Beutner & Pechuel (2012); Quentin-Baxter et al. (2008 

Better student performance 

Bonk et al. (2002); Quentin-Baxter et al. (2008); §ahin (2010a); Sancho et al. 
(2009); Singh (2003) 

Cost effectiveness 

Bastiaens & Martens (2000); Beutner & Pechuel (2012); Bollinger et al. (2011); 
Brennan (2004); Cohen & Nachmias (2009); Kim et al. (2008); Kim et al. (2009); 
Quentin-Baxter et al. (2008); Schooley (2009); Singh (2003) 

Expanding access to training 

Bastiaens & Martens (2000); Bollinger et al. (2011); Singh (2003) 

Immediacy, access just-in-time or 
in context 

Bastiaens & Martens (2000); Beutner & Pechuel (2012); Bollinger et al. (2011); 
Bonk et al. (2002); §ahin (2010b) 

Immediate feedback; interaction 

Bonk et al. (2002); Singh (2003) 
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Improved collaboration and 

teamwork 

Bonk et al. (2002); §ahin (2010b); Sancho et al. (2009 

Improved learning or instruction 

Bollinger et al. (2011); Cohen & Nachmias (2009); Kim et al. (2008); Kim et al. 
(2009); Quentin-Baxter et al. (2008); Singh (2003) 

Increased motivation; reduction of 
absences or drop-out rates 

Cohen & Nachmias (2009); Quentin-Baxter et al. (2008); §ahin (2010b); Singh 
(2003) 

Independence of location and time, 
flexibility, convenience 

Bastiaens & Martens (2000); Beutner & Pechuel (2012); Bollinger et al. (2011); 
Bonk et al. (2002); Brennan (2004); Kim et al. (2008); Kim et al. (2009); §ahin 
(2010b); Schooley (2009) 

Influence on organization: 

development, productivity, 

recruitment, competence 

Bastiaens & Martens (2000); Cohen & Nachmias (2009); Kim et al. (2008); Leary 
& Berge (2007); Quentin-Baxter et al. (2008); Schooley (2009) 

More active students 

Bollinger et al. (2011); Bonk et al. (2002); §ahin (2010b); Sancho et al. (2009) 

Repeatability and consistency 

Bollinger et al. (2011); Bonk et al. (2002); Schooley (2009) 

Time savings; better use of time 

Beutner & Pechuel (2012); Leary & Berge (2007); Singh (2003) 


Several trends emerge in the studies. Freedom from the limitations of time and place is appreciated as 
well as the possibility to learn “just-in-time” when the information is needed, possibly in context. Blended 
learning also offers the possibility to tailor and personalize the learning experience. Cost-effectiveness is 
another key factor, and it comes through both savings and improved processes. Blended learning is seen to be 
motivating for the student, and often results in better retention, quicker learning and improved results. 
Communication and interaction are improved, and students feel less self-conscious about asking questions or 
having to reread the material several times than they would in a traditional classroom situation. 


4. FINANCIAL IMPACT OF BLENDED LEARNING PROGRAMS 

Finding specific company cases proved to be difficult, probably because their sensitive nature makes 
companies reluctant to share them. Some companies do not even measure the impact their learning solutions 
have; in a study by Bonk (2002), nearly 60 % of the respondents’ companies had not completed a formal 
evaluation of the impact of Web-based learning. Of the companies that had measured the impact, 30 % had 
done an analysis such as the ROI calculation and 47 % had analysed the improvement in job performance. 

Wenger & Ferguson (2006) write that initial investments in the infrastructure in blended learning are 
high, but after the investments have been made, the cost per student tends to be lower. The cost of content is 
high as well, but after a while the increased multipurpose use lowers the cost for additional applications. 

Bersin (2004) points out that one should not roll out an e-leaming or blended learning program just to 
save costs. The initial investments of e-leaming solutions are usually quite high, comprising of infrashucture, 
content development and technology upgrades among other things. Also, savings can be a one-off thing, and 
a company cannot expect to continue on saving money on a yearly basis. According to Bersin, one should not 
measure the ROI of a training program based on how much money the company has saved. Instead, the focus 
should be on the improved skills that bring about new savings. Bersin states that if one increases productivity 
by one percent, it has over 10 times the financial impact of a one percent decrease in training costs. For 
companies wanting to save money, it is very important to focus on developing a purposeful, thought-out 
blended learning program - otherwise the initial costs will far outweigh the potential savings. 

Microsoft has published some figures of learning examples with Microsoft learning products. According 
to their case studies, blended solutions helped to lower the costs of training, citing reduction figures of 30 % 
in one case. In the case of Microsoft de Argentina, the blended solution cost “a fraction” of the corresponding 
face-to-face training. (Ziob & Mosher, 2006) 

At Avaya, a blended learning program helped salespersons perform better and increase the close rate, 
broaden the selection of products sold, increase the return on sales, reduce discounting and shorten the sales 
cycle. For the first six months after the learning program, the company reported over $36.3 million in 
incremental revenue with a 46.1 % ROI. (Chute et ah, 2006) 

Lewis & Orton (2006) reported an ROI of 17 to 1 for a learning program at IBM. Their calculations are 
based on the total costs for the creation and deployment of a management training module and the tangible 
cost benefits based on the use over 1 8 months. In addition to the cost savings, graduates of the program were 
asked to assess the direct financial first-year annual impact that the leadership training had on their 
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department. The average value given was $415,000, which gives an ROI figure of 47 to 1. IBM has also 
noticed that giving employees meaningful opportunities to learn and develop their skills means that they are 
less likely to leave the company in the first three years after they have been hired (DeViney & Lewis, 2006). 
As hiring and training new employees is expensive, improved employee retention can save a lot of money. 

Similar findings are reported in Worthen’s article (2001); the relatively high initial training investment of 
a company paid off because turnover was reduced through employee satisfaction. At GE, an ROI calculation 
on an orientation course revealed that employees could master the contents of the course in only three hours 
when studying online, whereas the onsite course had taken three days - and those three days required 
attendants to be flown to an offsite training facility. In this case, a lot of money was saved, especially since 
the training facility would have needed to be upgraded, causing an additional cost of $4.5 million. 

Schooley (2009) reminds that technologically enhanced learning environments are cost-effective once the 
technology is in place and the company culture is adjusted so that these new learning patterns are embraced. 
Relying too much on financial justification may endanger the effectiveness of the learning program. Self- 
paced e-leaming programs may enable learners to assimilate material often much faster than in a traditional 
classroom. 

Forrester have calculated the ROI for a hypothetical North American insurance company with 5,000 
employees (Schooley 2009). According to their calculations, the implementation of an e-leaming program 
would generate an estimated 69 % ROI over three years. The figures depend on how well the company 
succeeds in supporting their employees in the change. In the Forrester example, the company would invest 
heavily to implement the e-leaming program. The expected benefits exceed the costs, however, because 
savings can be gained from faster competency, training flexibility and consistency, and travel expenses. The 
Forrester model estimates that the investment will have paid itself back in 12 months. 

Kapp & Vasta (2003) have written about the methods for calculating a performance -based ROI. They also 
stress that many factors need to be taken into account to fully evaluate the financial impact of an e-leaming 
initiative. Although e.g. travel expenses can be cut and savings made, they do not really constitute an ROI 
and focus on the company’s main concern - the need for education. To really convince a company of a good 
return on their investment, it is necessary to show how their blended learning contribution will benefit the 
company financially in the long mn. Benefit types include improved employee productivity, improved 
quality of work and improved customer satisfaction. 

Bollinger et al. (201 1) have a case study on a HIV care course in Zambia. In their study, the online course 
initially cost $1,204 per student, but subsequent sessions were reduced to $171 per student, as the material 
was ready. The result was that the cost per student was significantly lower than that of comparable courses. 
In resource-limited and remote areas, distance learning programs are also less expensive to deploy. 

Singh (2003) concludes that while producing a Web-based self-paced and rich learning environment 
might be expensive to deploy, it could be just as effective to create a more affordable blend using generic off- 
the-shelf products, recorded events, case studies etc. It is not always a question of how fancy the system is, 
rather than whether it fits its purpose. In many cases, simpler solutions will do and thus enable savings. 

Beutner and Pechuel (2012) studied the acceptance of m-leaming in German companies. Their study 
showed that many people were concerned about the costs involved with the technology and could rarely see 
any real potential to make savings when compared to the current education budget. To make m-leaming more 
acceptable and attractive in companies, it would have to be seen as a money saving solution. 

Cohen & Nachmias (2009) developed a cost-effectiveness model for Web-supported insfruction that has 
been implemented at Tel Aviv University. The model uses data on how students and faculty use the Web, 
gathered from Web logs. The model provides information on the advantages that result from the 
improvement of the teaching process, not just financial benefits. An analysis of all the courses at the 
university revealed that a few courses managed to gather a significant part of the student benefits. This 
confirms that blended or e-leaming is a highly context-sensitive method and cannot guarantee equal results in 
every case, even in the same university. 

Brennan (2004) writes that it is difficult to clearly show a cause-and-effect relationship between training 
and performance, since it depends on so many different factors. It is also not possible to affect all areas of 
business improvement with just training. Many participants in Brennan’s study had difficulty in ascertaining 
the benefits of their blended learning programs through a ROI or cost-benefit analysis. Isolating the effects of 
training and finding all the necessary information is challenging, which makes doing the calculations harder. 

Leary & Berge (2007) write about the challenges smaller companies might face when considering e- 
leaming solutions. Small organizations might not have personnel, whose (sole) responsibility is to be in 
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charge of employee learning and training, but focus is needed to build a successful blended learning program. 
Small companies find it hard to justify the sometimes high initial investment costs of e-leaming, since the 
economies of scale do not apply to them, and the staff might have very different needs when it comes to 
training. According to Leary and Berge, the solution is to have a good strategy, one good training 
professional and, importantly, to start small. Even smaller companies can benefit from blended learning, and 
even a small effort can create a favourable outcome, if it is executed well. 

There is a growing trend in workplace learning to measure the impact of blended learning by evaluating 
the increase in learner skills. A study by Kim et al. (2009) revealed that their survey respondents predicted a 
trend toward assessing the impact of blended learning at a higher level. This would mean that the quality of 
blended learning would also be analysed through its benefits to the company using tools like ROI or cost- 
benefit analysis. (Kim et ah, 2008; Kim et ah, 2009) 


5. CONCLUSIONS 

The results of this literature study show that blended learning can offer many benefits for companies looking 
to diversify their learning portfolio. Blended learning enables learners to study whenever and wherever it 
suits their needs, often just when they need a particular piece of information. Blended learning also offers the 
possibility to review and repeat learning modules and possibly feel less self-conscious about doing so or 
asking questions. In an optimal case, the learning material is tailored, learners are motivated and the result is 
a more efficient and capable employee for the company. Increased employee performance can in the long run 
also lead to significant cost savings. 

In many cases, cost reductions can be the reason why a blended learning program is launched. In a 
simplified case, its reason for being is to eliminate costs related to travel, lodging and instructor time. These 
expectations can, however, lead to a huge disappointment. In order to build a successful blended learning 
program, substantial initial investments in technology are required. Also, careful work and planning goes into 
creating the right blend and relevant material. The material and the technology need to be updated regularly. 
This is by no means inexpensive, so it is important not to get blinded by the possibility of immediate cost 
savings. On the other hand, it is possible to benefit financially if the learning program is well-designed and 
will eventually lead to improvements in employee productivity or money-saving changes in organizational 
culture. Success depends largely on user acceptance and motivation and finding the right blend. Even if the 
result does not produce huge cost reductions, other benefits might still make it worthwhile. 

M-leaming solutions have gradually become notable elements of many blended learning programs. 
Therefore the impact analyses carried out in the field of blended learning are relevant also for the m-leaming 
research community. Use of the same terminology and measures enhances the collaboration between the 
research fields and makes it easier to develop the right blend based on diverse approaches. In addition, when 
proceeding with the m-leaming solutions from piloting to commercialization, it is increasingly important to 
be able to present measurable benefits. The models developed for assessing the impacts of the blended 
learning programs could be useful for that purpose. 
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